Standards and Practices for Forecasting

One hundred and thirty-nine principles are used to summarize knowledge about forecasting. They cover formulating a problem, obtaining information about it, selecting and applying methods, evaluating methods, and using forecasts. Each principle is described along with its purpose, the conditions under which it is relevant, and the strength and sources of evidence. A checklist of principles is provided to assist in auditing the forecasting process. An audit can help one to find ways to improve the forecasting process and to avoid legal liability for poor forecasting. Comments Reprinted with permission. Published in Principles of Forecasting: A Handbook for Researchers and Practitioners, J. Scott Armstrong (ed.): Norwell, MA: Kluwer Academic Publishers, 2001. This book chapter is available at ScholarlyCommons: http://repository.upenn.edu/marketing_papers/135 Principles of Forecasting: A Handbook for Researchers and Practitioners, J. Scott Armstrong (ed.): Norwell, MA: Kluwer Academic Publishers, 2001 Standards and Practices for Forecasting J. Scott Armstrong The Wharton School, University of Pennsylvania ABSTRACT One hundred and thirty-nine principles are used to summarize knowledge about forecasting. They cover formulating a problem, obtaining information about it, selecting and applying methods, evaluating methods, and using forecasts. Each principle is described along with its purpose, the conditions under which it is relevant, and the strength and sources of evidence. A checklist of principles is provided to assist in auditing the forecasting process. An audit can help one to find ways to improve the forecasting process and to avoid legal liability for poor forecasting. When managers receive forecasts, they often cannot judge their quality. Instead of focusing on the forecasts, however, they can decide whether the forecasting process was reasonable for the situation. By examining forecasting processes and improving them, managers may increase accuracy and reduce costs. One can examine the forecasting processes by systematically judging it against the 139 forecasting principles presented. These principles, organized into 16 categories, cover formulating problems, obtaining information, implementing methods, evaluating methods, and using forecasts. Why do you need 139 principles? You will not need all of them in any one situation. Nearly all of the principles are conditional on the characteristics of the situation. It would be misleading to write a book on “The Five Principles Used by Successful Forecasters.” They could never be appropriate for all the different situations that can arise. The principles were drawn primarily from the papers in Principles of Forecasting. They include the major principles, but ignore some that are specific only to a certain forecasting method. To help ensure that the principles are correct, this paper was subjected to extensive peer review over a period of three years. The paper was also posted in full text on the Forecasting Principles website with a plea for comments. There were over 36,000 visitors to the site during the three years and helpful suggestions were received. Twenty experts provided careful reviews, and suggestions were obtained when versions of the paper were presented at five academic conferences. I describe the strength of evidence for each principle and provide sources of empirical evidence. Many of the forecasting principles are based on expert opinion. I use the term “common sense” when it is difficult to imagine that things could be otherwise. “Received wisdom” indicates that the vast majority of experts agree.One hundred and thirty-nine principles are used to summarize knowledge about forecasting. They cover formulating a problem, obtaining information about it, selecting and applying methods, evaluating methods, and using forecasts. Each principle is described along with its purpose, the conditions under which it is relevant, and the strength and sources of evidence. A checklist of principles is provided to assist in auditing the forecasting process. An audit can help one to find ways to improve the forecasting process and to avoid legal liability for poor forecasting. When managers receive forecasts, they often cannot judge their quality. Instead of focusing on the forecasts, however, they can decide whether the forecasting process was reasonable for the situation. By examining forecasting processes and improving them, managers may increase accuracy and reduce costs. One can examine the forecasting processes by systematically judging it against the 139 forecasting principles presented. These principles, organized into 16 categories, cover formulating problems, obtaining information, implementing methods, evaluating methods, and using forecasts. Why do you need 139 principles? You will not need all of them in any one situation. Nearly all of the principles are conditional on the characteristics of the situation. It would be misleading to write a book on “The Five Principles Used by Successful Forecasters.” They could never be appropriate for all the different situations that can arise. The principles were drawn primarily from the papers in Principles of Forecasting. They include the major principles, but ignore some that are specific only to a certain forecasting method. To help ensure that the principles are correct, this paper was subjected to extensive peer review over a period of three years. The paper was also posted in full text on the Forecasting Principles website with a plea for comments. There were over 36,000 visitors to the site during the three years and helpful suggestions were received. Twenty experts provided careful reviews, and suggestions were obtained when versions of the paper were presented at five academic conferences. I describe the strength of evidence for each principle and provide sources of empirical evidence. Many of the forecasting principles are based on expert opinion. I use the term “common sense” when it is difficult to imagine that things could be otherwise. “Received wisdom” indicates that the vast majority of experts agree. Forecasters often ignore common sense and received wisdom. This observation was reinforced when I presented early versions of the principles to practitioners at the International Association of Business Forecasters in Philadelphia in 1997 and to academics at the “Judgmental Inputs to the Forecasting Process” conference at University College London in 1998. At both meetings, respondents to a questionnaire agreed with a vast majority of the principles, but they reported that few of these principles were followed in practice. PRINCIPLES OF FORECASTING 2 FORMULATING THE PROBLEM 1. Setting Objectives Specify the objectives in the situation, then consider what decisions relate to reaching those objectives. The issues in this section can help to decide whether it is worthwhile to use formal procedures to make forecasts. 1.1 Describe decisions that might be affected by the forecasts. Description: Analysts should examine how decisions might vary depending on the forecast. Purpose: To improve the use of forecasts by relating them to decision making. Conditions: Forecasts are needed only when they may affect decision making. If there are no decisions to be made, then there is no economic justification to do forecasting. Or, if the decision has already been made and cannot be changed, there is no need to make forecasts. Ignore this principle if the forecasts are strictly for entertainment, as with election-night forecasts. Strength of evidence: Common sense. Source of evidence: See Fischhoff (2001) for related evidence. 1.2 Prior to forecasting, agree on actions to take assuming different possible forecasts. Description. One approach is to ask decision makers to describe what forecasts will change their decisions. Another is to present alternative possible forecasts and ask what decisions they would make. For example, “If the forecast is less than 100, we cancel the project. If it is between 100 and 149, we get more information. If it is 150 or more, we continue.” Griffith and Wellman (1979) showed that independent quantitative forecasts of bed needs, obtained without prior agreement about how to use them, were ignored by hospital administrators when the forecasts were not to their liking. Purpose: To improve the use of forecasting. Conditions: Forecasts are needed only when they may affect decision making. Strength of evidence: Common sense. Source of evidence: None. 1.3 Make sure forecasts are independent of politics. Description. Separate the forecasting process from the planning process. One possibility is to have one group do the forecasting and another do the planning. Separating these functions could lead to different reports such as ones showing forecasts for alternative plans. This principle is sensible and important, yet it is often ignored. This is not surprising. Consider, for example, that you received a forecast that the passage of right-to-carry gun laws in the U.S. would have beneficial effects, such as reduced deaths. Would you consider that forecast in deciding how to vote on this issue? Purpose: To improve the use of forecasts by reducing bias. Conditions: Impartial forecasts are especially important when they imply major changes. Strength of evidence: Common sense. STANDARDS AND PRACTICES FOR FORECASTING 3 Source of evidence: Fildes and Hastings (1994), Griffith and Wellman (1979), Harvey (2001), Larwood and Whittaker (1977), and Sanders and Ritzman (2001). 1.4 Consider whether the events or series can be forecasted. Description. Would using formal forecasting procedures produce better forecasts than the current procedure or a naive benchmark? For example, short-term forecasts of the stock market do not improve accuracy (unless they are based on inside information). Purpose: To reduce costs by avoiding useless forecasting efforts. Conditions: When prior research shows that an area is unlikely to benefit, avoid formal forecasting. Use it, however, when formal forecasting produces accuracy equivalent to the current method but at a lower cost. Strength of evidence: Strong empirical support. Source of evidence: Much evidence shows that forecasters cannot beat the stock market with respect to accuracy. This goes as far back as Cowles (1933) and has continued ever since. 1.5 Obtain decision makers’ agreement on methods. Description. Describe how the forecasts are to be made, and do so in intuitive terms. Do the decision makers agree that they make sense? It may help to propose using a forecasting method on an experimental basis. Purpose: Agreement can improve the use of forecasts. Acceptance of the forecasts is more likely if decision makers believe the procedures are relevant. Conditions: The decision makers’ acceptance of forecasting methods is important when they control the use of the forecasts. Strength of evidence: Some empirical evidence. Source of evidence: Research on organizational behavior supports this principle, and implementing this principle proved useful in a laboratory experiment (Armstrong 1982). 2. Structuring the Problem The problem should be structured so the analyst can use knowledge effectively and so that the results are useful for decision making. 2.1 Identify possible outcomes prior to making forecasts. Description. Brainstorming about possible outcomes helps in structuring the approach. For example, experts might be asked to brainstorm the possible outcomes from the imposition of an affirmative action plan in a workplace. Purpose: To improve accuracy. Conditions: Determining possible outcomes is especially important for situations in which the outcomes are not obvious or in which a failure to consider a possible outcome might bias the forecast. Strength of evidence: Indirect evidence. Source of evidence: Tiegen (1983) shows how the specification of outcomes can affect predictions. For example, as new outcomes are added to a situation, forecasters often provide probabilities that exceed 100 percent. Arkes (2001) summarizes evidence relevant to this issue. PRINCIPLES OF FORECASTING 4 2.2 Tailor the level of data aggregation (or segmentation) to the decisions. Description. Decision makers should help to determine the need for forecasts specified by time, geography, or other factors. One can make forecasts, however, for various components that can then be aggregated or disaggregated to fit the decision needs. Thus, the analyst can focus on the level of aggregation that yields the most accurate forecasts. Purpose: To improve the use of forecasts by tailoring them to decisions. Conditions: Sufficient data must exist to enable different levels of aggregation. Strength of evidence: Common sense. Source of evidence: None. 2.3 Decompose the problem into parts. Description. Use a bottom-up approach. That is, forecast each component, then combine them. Purpose: To improve forecast accuracy by improving reliability. Also, by decomposing the problem, you can more effectively use alternative sources of information and different forecasting methods. Conditions: It is helpful to decompose the problem in situations involving high uncertainty and extreme (very large or very small) numbers. Additive breakdowns are preferable to multiplicative ones if the components = errors are highly correlated. Disaggregation will not improve accuracy if the components cannot be measured reliably. Decomposition by multiplicative elements can improve accuracy when you can forecast each of them more accurately than the target value. Strength of evidence: Received wisdom and strong empirical evidence. Source of evidence: Armstrong (1985) cites many studies. Evidence is also provided by Armstrong, Adya and Collopy (2001), Harvey (2001), and MacGregor (2001). 2.4 Decompose time series by causal forces. Description: Causal forces represent the expected directional effects of the factors that affect a series. They can be classified into the following categories: growth, decay, opposing, regressing, and supporting. Decompose by force, make extrapolations of the components, then synthesize the overall forecast. Purpose: To improve forecast accuracy. Conditions: Decompose by causal forces when time series are affected by factors that have conflicting effects on the trends and when they can be decomposed according to the type of causal force. This procedure can also be used for judgmental forecasts. Strength of evidence: Weak empirical evidence. Source of evidence: Burns and Pearl (1981) were unsuccessful in an attempt to use causal reasoning in helping experts decompose a forecasting problem. Armstrong, Adya and Collopy (2001) found that such decomposition improved accuracy in extrapolation. 2.5 Structure problems to deal with important interactions among causal variables. Description: Interactions imply that the relationship of X1 to Y is related to the level of X2. Purposes: To improve forecast accuracy; to assess effects of policy variables. STANDARDS AND PRACTICES FOR FORECASTING 5 Conditions: When interactions have important effects, you should account for them in the analysis. Though decomposition requires large samples, it provides a simple way to handle interactions. Strength of evidence: Received wisdom. Source of evidence: Little research has been done on this issue. However, in a study of sales at 2,717 gas stations, Armstrong and Andress (1970) found that decomposition to handle interactions substantially improved accuracy in comparison with forecasts from a regression model. 2.6 Structure problems that involve causal chains. Description: Given a series of effects such as X causes Y, which then causes Z, simultaneous equations have not led to improved accuracy. Instead, construct a series of linked models. That is, develop a model using X to predict Y. Then use the predictions for Y, in a model to predict Z. Purpose: To improve accuracy. Conditions: Use causal chains when they have important effects, their relationships are well known, and the timing can be accurately forecast. Strength of evidence: Received wisdom and some empirical evidence. Source of evidence: Allen and Fildes (2001); Armstrong (1985, pp. 199-200). 2.7 Decompose time series by level and trend. Description: The separate examination of level and trend is one of the oldest and more enduring principles, and it is widely used in practice. Purpose: To improve forecast accuracy. Conditions: Decomposition is useful when there are significant trends that can be assessed by different methods. For example, judgmental methods are especially useful for incorporating recent information into estimates of levels. Strength of evidence: Received wisdom and some empirical evidence. Source of evidence: Armstrong (1985, pp. 235-238) summarizes evidence from eight studies. OBTAINING INFORMATION This section examines the identification, collection, and preparation of data to be used in forecasting. 3. Identify Data Sources Identify data that might be useful in making forecasts. While this should be guided by theory, you may need creativity in seeking alternative types of data. 3.1 Use theory to guide the search for information on explanatory variables. Description: Theory and prior research can help in the selection of data on explanatory variables. For example, in sales forecasting, a common model is to predict sales based on market size, ability to purchase, and need. The search for information could then be limited to these variables. Operational measures are then needed – such as income, availability, and price – to measure “ability to purchase.” Purpose: To improve forecast accuracy. PRINCIPLES OF FORECASTING 6 Conditions: To follow this principle, analysts must have good prior knowledge. That knowledge can be based on experience or research studies. Strength of evidence: Received wisdom with little empirical testing. Received wisdom has been questioned by practitioners who violate this principle in the belief that more information is always better. Some researchers have ignored this principle in favor of data mining, which assumes that the data will reveal causal patterns. Source of evidence: Armstrong (1985, pp. 52-57) describes studies that show how one can get absurd results by ignoring theory. It also describes a study in which a theory-based econometric model was more accurate than a model based only on statistical criteria. 3.2 Ensure that the data match the forecasting situation. Description: Data about past behavior in that situation are often the best predictors of future behavior. Purpose: To improve forecast accuracy. Conditions: This principle applies to all conditions, but especially when it is not obvious which data you should use to match the situation. Strength of evidence: Strong empirical support from research in personnel selection. Source of evidence: Armstrong (2001a,f) and Morwitz (2001) summarize some of the evidence. For example, studies have shown that personnel selection tests should be similar to the job requirements. 3.3 Avoid biased data sources. Description: Avoid data collected by persons or organizations that are obviously biased to particular viewpoints, perhaps because they are rewarded for certain outcomes. Thus, for extrapolating crime rates, victim surveys would be preferable to police records. Identify biases before analyzing the data, especially when people are emotional about the outcomes, as in forecasting the effects of environmental hazards. Consider this forecast made by the biologist Paul Ehrlich, on the first Earth Day on April 22, 1970: “Population will inevitably and completely outstrip whatever small increases in food supply we make.” Purpose: To improve accuracy. Conditions: Follow this principle when you can identify biased sources and alternative sources of data are available. Strength of evidence: Common sense. Source of evidence: None. 3.4 Use diverse sources of data. Description: Find alternative ways of measuring the same thing. If unbiased sources are not available, find sources with differing (and hopefully compensating) biases. For example, exports of a product from country A to country B should equal imports of that product to country B from country A. If the alternative sources do not agree, consider combining estimates from each source. Purpose: To improve forecast accuracy. Conditions: Use diverse sources when biases are likely to occur. Strength of evidence: Received wisdom with some empirical support. Source of evidence: Armstrong (1985, p. 236) mentions two studies. STANDARDS AND PRACTICES FOR FORECASTING 7 3.5 Obtain information from similar (analogous) series or cases. Such information may help to estimate trends. Description: Trends in analogous time series may provide useful information for trends in the series of interest. For example, the trendline for sales of all luxury cars might help to estimate the projected trend for a specific brand of luxury car. Purpose: To improve forecast accuracy. Conditions: You must be able to identify similar data. Analogous data are especially important when the series of interest has few observations or high variability. Strength of evidence: Received wisdom with little empirical support. Source of evidence: Duncan, Gore and Szczpula (2001) provides some evidence for time series. Claycamp and Liddy (1969) provide evidence from their study on sales forecasts for new products.